Patch description

If a job is pre-empted and then re-queued, the current logic produces multiple runs in WandB, each corresponding a respective execution of the job. We should tell WandB to automatically group runs of the same job into a single run if a process exited unsuccessfully (i.e. was pre-empted).

For more detail, see: https://docs.wandb.ai/guides/track/advanced/resuming.

Testing steps

I ran the following command twice, Ctrl+C'ing after the first one, using my WandB account to verify that it did not produce two runs, but instead joined the logs from both runs. If you want to run this yourself, you need to login to your WandB account and update --wandb-entity arg to your username below.

parlai train_model -t personachat -m transformer/ranker --n-layers 1 --embedding-size 300 --ffn-size 600 --n-heads 4 --num-epochs 10 -veps 0.25 -bs 64 -lr 0.001 --dropout 0.1 --embedding-type fasttext_cc --candidates batch -wblog True --wandb-name a_test --wandb-project test2 --wandb-entity colinflaherty --model-file './tmp/model'

This is what things look like if for the code currently on main:

This is what things look like with this PR's changes added: